Repository navigation
[Klaud Cold] minimaxm31flash-fp4-gb300-sglang-agentic-dspark: day-zero MiniMax-M3.1-Flash GB300 SGLang recipe - #4443
Oseltamivir wants to merge 3 commits into
Conversation
…1-Flash SGLang AgentX on GB300 Serve flags follow the SGLang cookbook's MiniMax-M3.1 cells with the day-zero dev-minimax-m31-flash image. Seven-token DSpark with synthetic acceptance from the minimaxm31flash_dspark golden curve.
|
Thanks for the contribution!
中文感谢你的贡献!
|
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39996924954 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39996924954 |
| args: | ||
| tensor-parallel-size: 4 | ||
| expert-parallel-size: 4 | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 128 | ||
| cuda-graph-max-bs-decode: 128 | ||
| mem-fraction-static: 0.8 | ||
| env: | ||
| SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768' |
There was a problem hiding this comment.
🔴 The conc 64 and 128 results will be published as High-Throughput MegaMoE numbers, but those runs never actually use MegaMoE. Neither EP4 override sets moe-a2a-backend: megamoe, so SGLang falls back to its default MoE path. That leaves SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK with no effect. Fix: add moe-a2a-backend: megamoe to the args of both EP4 overrides (and the MoE runner the cookbook's High-Throughput cell uses), which covers the 2 sites listed. Same pattern at 2 sites (agentic.yaml:135, agentic.yaml:152).
Why this was flagged
The matrix point { tp: 4, ep: 4, ... conc-list: [64, 128] } at nvidia-master.yaml binds to override_tp4_ep4_c64 and override_tp4_ep4_c128 (agentic.yaml:132, agentic.yaml:149). Their args set only tensor-parallel-size: 4 and expert-parallel-size: 4, with no moe-a2a-backend and no moe-runner-backend. Every other SGLang MegaMoE recipe in the repo sets moe-a2a-backend: megamoe next to this same env var, for example dsv41flash/sglang/gb200-fp4-mtp/agentic.yaml:189 with :196, and dsv4/sglang/b200-fp4-mtp/agentic.yaml:232 with :246. So the server starts with the default a2a backend, and the DeepGEMM MegaMoE token cap at agentic.yaml:143 and :160 is never read. Nothing in the binder adds that flag, so the conc 64/128 results land in the changelog labelled 'TP4 + EP4 (MegaMoE)' even though they come from a different MoE path.
Verification: The args of override_tp4_ep4_c64 (agentic.yaml:132-147) and override_tp4_ep4_c128 (:149-164) set no moe-a2a-backend and no moe-runner-backend, and neither does the base block. The only MegaMoE-related setting is the env var SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768' (:143, :160). Every other SGLang recipe that sets this env var also turns MegaMoE on explicitly.
| - "Serve flags follow the cookbook's GB300 Balanced (TP4) and High-Throughput (TP4, EP4): DSpark with block verification, breakable prefill graphs, minimax-m3 reasoning and tool parsers, 8 tokenizer workers." | ||
| - "Seven-token DSpark (the bundled draft's dspark_block_size); throughput runs use synthetic acceptance from the minimaxm31flash_dspark golden curve (AL 4.04 at 7 draft tokens, #4425)." | ||
| - "Search space: Balanced TP4 at conc 1-32; TP4 + EP4 at conc 64-128." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The changelog check on this PR will fail, and so will the merge-prep step that rewrites the link. The new block uses pr-link: TBD, which is not an accepted placeholder. PR_LINK_PLACEHOLDERS in validate_perf_changelog.py:22 only allows XXX or https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. Fix: set the line to pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX or to this PR's canonical URL. Both validators accept either form.
Why this was flagged
The appended entry at perf-changelog.yaml:11018 has pr-link: TBD. On a PR run, compare_entries calls validate_added_pr_link (validate_perf_changelog.py:193). That function raises new PR entry must use '<url>' or an XXX placeholder; found 'TBD' (line 147-149) because TBD is not in PR_LINK_PLACEHOLDERS (line 22-25). On main, it fails CANONICAL_PR_LINK with new main-branch entry has invalid pr-link: 'TBD'. prepare_perf_changelog_merge.py:108-110 also refuses with appended entry N has unexpected pr-link 'TBD', so the merge helper cannot swap in the real link. No other entry in the file uses TBD. The changelog validation job fails, and result collection for minimaxm31flash-fp4-gb300-sglang-agentic-dspark is blocked until the link is fixed.
Verification: Trigger: the PR goes through the reuse-assisted merge with the pr-link still TBD. merge_with_reuse.py:631 always calls canonicalize_changelog, which reaches validate_added_pr_link (validate_perf_changelog.py:193). Since TBD is not in PR_LINK_PLACEHOLDERS (lines 22-25), it raises, so merge prep stops with an error. The PR-run CI check does not fail.
| override_tp4_ep4_c128: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| expert-parallel-size: 4 | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 128 | ||
| cuda-graph-max-bs-decode: 128 | ||
| mem-fraction-static: 0.8 |
There was a problem hiding this comment.
🟡 (optional) The conc 128 point admits only 128 requests, 1x CONC. Subagent fan-out above that waits in the SGLang queue and is measured as TTFT. Every other point here, and the vLLM sibling at c128, admits 2x CONC. In override_tp4_ep4_c128, max-running-requests: 128 caps admission at CONC, while c64 with the same server admits 2x. Fix: admit 2x CONC (256) at c128 and keep cuda-graph-max-bs-decode: 128, as the dsv41flash GB300 SGLang DEP2 c128 variant does, or drop the c128 point. 256 x 8 verify tokens is far below the 32768-token MegaMoE per-rank budget.
Why this was flagged
The matrix point { tp: 4, ep: 4, ... conc-list: [64, 128] } (nvidia-master.yaml:8896) binds c128 to override_tp4_ep4_c128, which sets max-running-requests: 128 and CONC: '128' (agentic.yaml:149-163). AgentX replays sessions at --concurrency CONC (infx/bench/agentic/replay.py:142). Sessions with subagents put more requests in flight than CONC. That is why this recipe's own comment (agentic.yaml:51-52) and its siblings size admission at 2x CONC. At c128, any burst above 128 in-flight requests waits in the scheduler's waiting queue, and the wait shows up in TTFT and end-to-end latency. c64 runs the same server with 2x headroom, so c64 and c128 are not measured under the same admission rule. The vLLM GB300 sibling DEP4 c128 admits 64 per rank x 4 = 256 (vllm/gb300-fp4-mtp/agentic.yaml), so the published SGLang c128 row is biased against SGLang. That cap is not a technical limit here.
Verification: In agentic.yaml, override_tp4_ep4_c128 sets max-running-requests: 128 together with CONC: '128' (lines 149-163). That admits only 1x CONC. In the same file, override_tp4_ep4_c64 keeps max-running-requests: 128 at CONC: '64', which is 2x. Requests above 128 wait in SGLang's queue, and that wait is counted as TTFT.
| # MiniMax-M3.1-Flash AgentX on GB300 with SGLang: day-zero image from the SGLang | ||
| # cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1), | ||
| # manifest digest verified 2026-10-24. Serve flags follow the cookbook's | ||
| # GB300 Balanced (TP4) and High-Throughput (TP4, EP4). | ||
| # Seven-token DSpark (bundled draft); synthetic acceptance follows | ||
| # infx/golden_al_distribution/minimaxm31flash_dspark.yaml. |
There was a problem hiding this comment.
🟡 (optional) This adds header and provenance comments that AGENTS.md's mandatory code-comment rules forbid, so later readers get history in place of technical notes. The class is section/header or cookbook-provenance comments on new lines. Fix: delete these comments and keep only one-line technical notes on specific values (the health_check comment is fine), which covers the 5 sites listed. Same pattern at 5 sites (nvidia-master.yaml:8875-8880, nvidia-master.yaml:8893, nvidia-master.yaml:8895, agentic.yaml:24-25, agentic.yaml:51-52 and 130-131).
Why this was flagged
AGENTS.md "Code comments (mandatory)" applies to recipes and configs. It says: no header comments above a recipe, override or section; no comments about "where it was derived from"; only short technical explanations. nvidia-master.yaml:8875-8880 is a block above the new key. It describes the config, cites the SGLang cookbook URL, and records "manifest digest verified 2026-10-24", which is provenance and a date after today's 2026-10-23. nvidia-master.yaml:8893 "# Balanced: TP4." and :8895 "# High-Throughput: TP4 + EP4 (MegaMoE)." are section labels that restate the row. The recipe has "Serve flags follow the cookbook's GB300 cells at " (agentic.yaml:24-25). It also has the section headers "# TP4 Balanced cell ..." (51-52) and "# High-Throughput cell ..." (130-131) above the override groups. The sibling vLLM GB300 recipe has none of these. Merging adds forbidden comment styles that the repo's own rules say belong in git history and perf-changelog.yaml.
Verification: AGENTS.md:60-66 "Code comments (mandatory)" says the rules apply to "every file, including recipes, configs". They forbid header comments "above a recipe, override, or section" and history comments about "where it was derived from". nvidia-master.yaml:8875-8880 is a header block above the new key. Nothing breaks at runtime.
Keep main's entries and re-add the GB300 SGLang changelog and master-config blocks.
|
replaced by #4456 |
Summary
Day-zero MiniMax-M3.1-Flash (
MiniMaxAI/MiniMax-M3.1-Flash, NVFP4) AgentX on GB300 with SGLang, following the SGLang cookbook: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1 (config source:docs/src/snippets/configs/MiniMaxAI/minimax-m3.1.jsx).lmsysorg/sglang:dev-minimax-m31-flash, pinned by digest (verified on Docker Hub 2026-10-24).--speculative-algorithm DSPARK --speculative-use-block-verification,--cuda-graph-backend-prefill breakable,minimax-m3reasoning and tool parsers,--tokenizer-worker-num 8.--speculative-dspark-block-size 7, the bundled draft'sdspark_block_size). It's set explicitly so throughput runs resolve synthetic acceptance fromminimaxm31flash_dspark.yaml([Klaud Cold] Add MiniMax-M3.1-Flash DSpark golden AL curve and SPEED-Bench collector / [Klaud Cold] 新增 MiniMax-M3.1-Flash DSpark 黄金 AL 曲线与 SPEED-Bench 采集脚本 #4425): AL 4.04 at 7 tokens, thinking on (adaptive, the default).infx generatebinds every matrix point to exactly one recipe variant, withSGLANG_SIMULATE_ACC_LEN=4.04.vLLM siblings: #4419 (GB300), #4420 (MI355X), #4421 (B300), #4422 (H100), #4423 (H200), #4426/#4431 (B200), #4433 (GB200).